Motion and gesture input from a wearable device

ABSTRACT

This disclosure relates to detecting hand gesture input using an electronic device, such as a wearable device strapped to a wrist. The device can have multiple photodiodes, each sensing light at a different position on a surface of the device that faces skin of a user. Examples of the disclosure detect hand gestures by recognizing patterns in sensor data that are characteristic of each hand gesture, as the tissue expands and contracts and anatomical features in the tissue move during the gesture.

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 62/233,295, filed Sep. 25, 2015.

FIELD OF THE DISCLOSURE

This relates generally to detecting a user's motion and gesture input toprovide commands to one or more devices. In particular, a device can useone or more sensors to determine a user's motion and gesture input basedon movements of the user's hand, arm, wrist, and fingers.

BACKGROUND OF THE DISCLOSURE

Some existing portable electronic devices accept voice or touch input tocontrol functionality of the devices. For example, a voice commandsystem can map specific verbal commands to operations such as initiatinga voice call with a particular contact in response to speaking thecontact's name. In another example, a touch input system can mapspecific touch gestures to operations such as zooming out in response toa pinch gesture on a touch sensitive surface. However, there may besituations where the user's ability to speak a verbal command or performa touch gesture may be limited.

SUMMARY OF THE DISCLOSURE

This disclosure relates to detecting hand gesture input using anelectronic device, such as a wearable device strapped to a wrist. Thedevice can have multiple photodiodes, each sensing light at a differentposition on a surface of the device that faces skin of a user. Due tothis positioning, the sensor data from the photodiodes can capturemovement of anatomical features in the tissue of the user during a handgesture. Further, different light emitters on the device can emit lightat different wavelengths (e.g., infrared light, green light, etc.),which penetrate to different depths in the tissue of the user beforereflecting back to the photodiodes on the device. Accordingly, sensordata from the photodiodes can capture expansion and contraction in thetissue of the user during a hand gesture. Examples of the disclosuredetect hand gestures by recognizing patterns in sensor data that arecharacteristic of each hand gesture, as the tissue expands and contractsand anatomical features in the tissue move during the gesture.

In one example, the device can be trained on sensor data as the userperforms a plurality of hand gestures. For example, during a firstperiod, a user can perform a hand flap gesture and sensor data can becollected as the gesture is performed. During a second period, a usercan perform a hand clench gesture and further sensor data can becollected as the gesture is performed. The sensor data can then beprocessed to calculate signal characteristics (e.g., peak/troughextraction, phase detection, etc., as described below) based on thesensor data for each period. The signal characteristics can then beclustered (e.g., using a clustering algorithm such as k-meansclustering), including assigning some or all of the signalcharacteristics from the first period to a first cluster and some or allof the signal characteristics from the second period to a secondcluster. The first cluster can be considered a pattern that ischaracteristic of the hand flap gesture that was performed during thefirst period, and the second cluster can be considered a pattern that ischaracteristic of the hand clench gesture that was performed during thesecond period.

Gesture detection can then be performed based, in part, on the clustersformed during training. Further sensor data can be collected during athird period, and signal characteristics can be calculated from thatsensor data. After calculating the signal characteristics from the thirdperiod, a hand gesture can be detected based on the cluster that thesignal characteristics belong to. For example, if most of the signalcharacteristics from the third period belong to the first cluster, thena hand flap gesture can be detected. If most of the signalcharacteristics from the third period belong to the second cluster, thena hand clench gesture can be detected. If most of the signalcharacteristics from the third period belong to a third cluster, then itcan be determined that the user has not performed the first hand gestureor the second hand gesture.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various described examples, referenceshould be made to the Detailed Description below, in conjunction withthe following drawings in which like reference numerals refer tocorresponding parts throughout the figures.

FIGS. 1A-1C illustrate an exemplary electronic device with a pluralityof sensors in accordance with examples of the disclosure.

FIGS. 2A-2D illustrate exemplary hand gestures in accordance withexamples of the disclosure.

FIG. 3 illustrates exemplary charts of sensor data in accordance withexamples of the disclosure.

FIGS. 4A-4D illustrate two-dimensional clustering examples in accordancewith examples of the disclosure.

FIGS. 5A-5B illustrate training user interfaces in accordance withexamples of the disclosure.

FIG. 6 illustrates an exemplary method of training for gesture detectionin accordance with examples of the disclosure.

FIG. 7 illustrates an exemplary method of gesture detection inaccordance with examples of the disclosure.

FIG. 8 is a block diagram illustrating a portable multifunction devicewith a touch-sensitive display in accordance with some examples.

DETAILED DESCRIPTION

In the following description, reference is made to the accompanyingdrawings which form a part hereof, and in which it is shown by way ofillustration specific examples that can be practiced. It is to beunderstood that other examples can be used and structural changes can bemade without departing from the scope of the disclosed examples.

This disclosure relates to detecting hand gesture input using anelectronic device, such as a wearable device strapped to a wrist. Thedevice can have multiple photodiodes, each sensing light at a differentposition on a surface of the device that faces skin of a user. Due tothis positioning, the sensor data from the photodiodes can capturemovement of anatomical features in the tissue of the user during a handgesture. Further, different light emitters on the device can emit lightat different wavelengths (e.g., infrared light, green light, etc.),which penetrate to different depths in the tissue of the user beforereflecting back to the photodiodes on the device. Accordingly, sensordata from the photodiodes can capture expansion and contraction in thetissue of the user during a hand gesture. Examples of the disclosuredetect hand gestures by recognizing patterns in sensor data that arecharacteristic of each hand gesture, as the tissue expands and contractsand anatomical features in the tissue move during the gesture.

In one example, the device can be trained on sensor data as the userperforms a plurality of hand gestures. For example, during a firstperiod, a user can perform a hand flap gesture and sensor data can becollected as the gesture is performed. During a second period, a usercan perform a hand clench gesture and further sensor data can becollected as the gesture is performed. The sensor data can then beprocessed to calculate signal characteristics (e.g., peak/troughextraction, phase detection, etc., as described below) based on thesensor data for each period. The signal characteristics can then beclustered (e.g., using a clustering algorithm such as k-meansclustering), including assigning some or all of the signalcharacteristics from the first period to a first cluster and some or allof the signal characteristics from the second period to a secondcluster. The first cluster can be considered a pattern that ischaracteristic of the hand flap gesture that was performed during thefirst period, and the second cluster can be considered a pattern that ischaracteristic of the hand clench gesture that was performed during thesecond period.

Gesture detection can then be performed based, in part, on the clustersformed during training. Further sensor data can be collected during athird period, and signal characteristics can be calculated from thatsensor data. After calculating the signal characteristics from the thirdperiod, a hand gesture can be detected based on the cluster that thesignal characteristics belong to. For example, if most of the signalcharacteristics from the third period belong to the first cluster, thena hand flap gesture can be detected. If most of the signalcharacteristics from the third period belong to the second cluster, thena hand clench gesture can be detected. If most of the signalcharacteristics from the third period belong to a third cluster, then itcan be determined that the user has not performed the first hand gestureor the second hand gesture.

Although examples of the disclosure may be described herein primarily interms of wearable devices strapped to a wrist and hand gestures,particularly hand flap gestures and hand clench gestures, it should beunderstood that examples of the disclosure are not so limited, butinclude wearable devices attached to other body parts, such as upperarms or legs, and gestures that can result therefrom.

Further, although examples of the disclosure may be described hereinprimarily in terms of devices with a plurality of photodiodes, it shouldbe understood that examples of the disclosure are not so limited, butinclude devices with only a single photodiode. A channel of sensor datacan correspond to each unique light sensor/emitter pair, whether thereis one or multiple sensors, one or multiple emitters, etc.

FIGS. 1A-1C illustrate an exemplary electronic device 100 with aplurality of sensors in accordance with examples of the disclosure. Theelectronic device 100 can include a plurality of photodiodes 101 (or anyother light sensors) and a plurality of light emitters 105 (e.g.,light-emitting diodes, etc.). When the electronic device is in use, thephotodiodes 101 and the light emitters 105 face the tissue 200 of auser.

As illustrated in FIG. 1B, each photodiode 101 can sense light at adifferent position on a surface of the device 100 that faces the tissue200 of a user. Due to this positioning, the sensor data from thephotodiodes 101 can capture movement of anatomical features in thetissue 200 of the user during a hand gesture. Further, as illustrated inFIG. 1C, different light emitters 105 on the device 100 can emit lightat different wavelengths (e.g., infrared light 600, green light 602,etc.), which penetrate to different depths in the tissue 200 of the userbefore reflecting back to the photodiodes on the device. Accordingly,sensor data from the photodiodes 101 can capture expansion andcontraction in the tissue 200 of the user during a hand gesture.

In some examples, each possible photodiode emitter combination can beconsidered a separate channel of light sensor data. For example, in adevice with two green light emitters, two infrared light emitters, andtwo photodiodes, there can be eight channels of light sensor data. Whenthe first green light emitter emits light, the first and secondphotodiodes sense first and second channels of light sensor data,respectively. When the second green light emitter emits light, the firstand second photodiodes sense third and fourth channels of light sensordata, respectively. When the first infrared light emitter emits light,the first and second photodiodes sense fifth and sixth channels of lightsensor data, respectively. When the second infrared light emitter emitslight, the first and second photodiodes sense seventh and eight channelsof light sensor data, respectively.

In some examples, the device 100 can further include sensors thatprovide additional channels of sensor data. For example, a device thatincludes an accelerometer and a force sensor can provide four additionalchannels of sensor data. A force sensor that detects force of the wristagainst the device can provide a first additional channel of sensordata. An accelerometer that senses acceleration in X, Y, and Zdirections can provide second, third, and fourth additional channels ofsensor data, respectively. In some examples, additional channels ofsensor data can include data from a barometer, a magnetometer, a GlobalPositioning System (GPS) receiver, and/or numerous other possibilities.In some examples, other light sensors are used in place of or inaddition to photodiodes. In some examples, a force sensor can bespatially discretized, sensing force independently at multiple positionsof the surface of the device that contacts the wrist, in which case theforce sensor can provide multiple (e.g., 4) channels of pressureinformation.

FIGS. 2A-2D illustrate exemplary hand gestures in accordance withexamples of the disclosure. In some examples, a hand 202 can perform ahand flap gesture, as illustrated in FIGS. 2A-2C. In FIG. 2A, thefingers of the hand 202 are fully extended. In FIG. 2B, the fingers comedown toward the palm of the hand 202. In FIG. 2C, the fingers return toa fully extended position.

In some examples, a hand 202 can perform a hand clench gesture, asillustrated in FIG. 2D. FIG. 2D illustrates the hand 202 in a firstformation. The hand clench gesture can be performed by clenching thehand 202 when in the first formation.

The FIGS. 2A-2D are examples of hand clench and hand flap gestures.Variations of these gestures and other gestures entirely may be trainedon and detected in accordance with examples of the disclosure.

FIG. 3 illustrates exemplary charts of sensor data in accordance withexamples of the disclosure. For example, the sensor data 300, 302, 304,and 306 can represent different channels of sensor data during a handclench gesture, and the sensor data 308, 310, 312, and 314 can representdifferent channels of sensor data during a hand flap gesture. In oneexample, 300 and 308 can represent sensor data from the first lightemitter/first photodiode pair (channel L1P1), 302 and 310 can representsensor data from the first light emitter/second photodiode pair (channelL1P2), 304 and 312 can represent sensor data from the second lightemitter/first photodiode pair (channel L2P1), and 306 and 314 canrepresent sensor data from the second light emitter/second photodiodepair (channel L2P2).

As can be observed in FIG. 3, during the clench gesture, channels L1P1,L2P1, and L2P2 each exhibit a signal characteristic wherein the troughoccurs just before the peak of the signal. In contrast, during the flapgesture, channels L1P2, L2P1, and L2P2 each exhibit a signalcharacteristic wherein the peak occurs just before the trough of thesignal. Accordingly, the two gestures can be distinguished based onsignal characteristics of the L2P1 and L2P2 channels.

Whereas the visual signal characteristics can be easily observed in theexemplary sensor data of FIG. 3, a number of quantitative signalcharacteristics may be calculated based on the sensor data beforeclustering. For example, an amplitude difference can be calculatedbetween a peak and a trough of the sensor data, with sign indicatingwhether the peak comes before the trough or vice versa, a timedifference can be calculated between a peak and a trough of the sensordata, a maximum amplitude can be calculated, a period between peaks ofthe sensor data can be calculated, and/or a phase can be detected in thesensor data, among other possibilities. In some examples, signalcharacteristics can be observed in a frequency domain. For example, oneor more frames of sensor data may be analyzed (e.g., by a Fouriertransform) to extract frequency information as additional signalcharacteristics. These and other signal characteristics can be extractedfrom any or all of the channels of sensor data, including light sensors,force sensors, an accelerometer, and/or other sensors.

FIGS. 4A-4D illustrate two-dimensional clustering examples in accordancewith examples of the disclosure. In some examples, each frame in sensordata collection can be considered a point in multi-dimensional space,with each calculated signal characteristic for that frame being acoordinate in the multi-dimensional space. For example, sensor data canbe collected during a first period in which a first gesture is performedby the user. The sensor data can be divided into a plurality of frames,and each frame can correspond to a set of coordinates defined by thesignal characteristics calculated for that time frame. The dataillustrated in FIGS. 4A-4D represent data collected with two signalcharacteristics: amplitude difference between peak and trough for theL2P1 and L2P2 channels, as discussed above with respect to FIG. 3.Although these figures only show two signal characteristics, examples ofthe disclosure are not so limited and contemplate using multiple kindsof signal characteristics from multiple channels, including light sensordata, force sensor data, and/or accelerometer data, among otherpossibilities.

FIG. 4A illustrates sensor data collected during a first period in whicha hand flap gesture is being performed as illustrated in FIGS. 2A-2C,possibly multiple times in succession. FIG. 4B illustrates sensor datacollected during a second period in which a hand clench gesture is beingperformed as illustrated in FIG. 2D, possibly multiple times insuccession. FIG. 4C illustrates sensor data collected during a thirdperiod in which no gesture is being performed. FIG. 4D illustrates thesensor data collected during all three periods, clustered into threeclusters: first cluster 400, second cluster 402, and third cluster 404.

As shown in FIGS. 4A-4D together, most of the points corresponding tothe first period belong to the first cluster, most of the pointscorresponding to the second period belong to the second cluster, andmost of the points corresponding to the third period belong to the thirdcluster. Accordingly, it may be inferred that any point that belongs tothe first cluster 400 was collected during performance of a hand flapgesture, and any point that belongs to the second cluster 402 wascollected during performance of a hand clench gesture. Further, it maybe inferred that any point that does not belong to the first cluster 400or the second cluster 402 was not collected during performance of a handclench or hand flap gesture.

FIGS. 5A-5B illustrate training user interfaces in accordance withexamples of the disclosure. FIG. 5A illustrates a user interfaceprompting a user to perform a hand flap gesture. Such a user interfacemay be displayed during a first period and sensor data may be collectedduring the first period while the user interface is displayed. FIG. 5Billustrates a user interface prompting a user to perform a hand clenchgesture. Such a user interface may be displayed during a second periodand sensor data may be collected during the second period while the userinterface is displayed. Additional user interfaces may be displayedprompting a user to perform additional gestures to train for detectionof the additional gestures.

FIG. 6 illustrates an exemplary method of training for gesture detectionin accordance with examples of the disclosure. In some examples, sensordata can be collected while the user performs various hand gestures totrain the detection algorithm. During a first period in which the userperforms a first hand gesture (e.g., when prompted by a user interfaceas illustrated in FIG. 5A), the electronic device can collect (600)first sensor data from the plurality of photodiodes. During a secondperiod in which the user performs a second hand gesture (e.g., whenprompted by a user interface as illustrated in FIG. 5B), the electronicdevice can collect (602) second sensor data from the plurality ofphotodiodes. In some examples, sensor data is further collected fromother sensors, such as a force sensor and/or an accelerometer, amongother possibilities.

After sensor data collection, signal characteristics can be extractedfrom the sensor data, as discussed above with respect to FIG. 3. Theelectronic device can calculate first signal characteristics (604) basedon the first sensor data and calculate second signal characteristics(606) based on the second sensor data.

After calculating the signal characteristics, the electronic device canperform (608) clustering (e.g., a k-means clustering algorithm or otherclustering algorithm) on the first and second signal characteristics.The clustering algorithm may assign (610) some or all of the firstsignal characteristics to a first cluster of signal characteristics, andassign (612) some or all of the second signal characteristics to asecond cluster of signal characteristics.

In some examples, the electronic device can assign each cluster to oneof the hand gestures as part of the training process. For example, theelectronic device can compare the first cluster to the second cluster.Then, based on comparing the first cluster to the second cluster, theelectronic device can determine there are more of the first signalcharacteristics assigned to the first cluster than to the secondcluster. In accordance with the determination that there are more of thefirst signal characteristics assigned to the first cluster than to thesecond cluster, the electronic device can assign the first cluster tothe first hand gesture. Similarly, based on comparing the first clusterto the second cluster, the electronic device can determine there aremore of the second signal characteristics assigned to the second clusterthan to the first cluster. In accordance with the determination thatthere are more of the second signal characteristics assigned to thesecond cluster than to the first cluster, the electronic device canassign the second cluster to the second hand gesture.

In some examples, the clustering process can be seeded by initiallyclustering the signal characteristics based on the time period in whichthe data was collected. For example, the first cluster can be initiallyassigned all the signal characteristics corresponding to the firstperiod during which the first hand gesture was performed, and the secondcluster can be initially assigned all the signal characteristicscorresponding to the second period during which the second hand gesturewas performed. Following this initial assignment, a clustering algorithm(e.g., k-means clustering) can be performed to optimize the clusters,potentially moving some points from the first cluster to the secondcluster, moving some points from the second cluster to the firstcluster, and/or moving some points from the first and second clusters toother clusters.

In some examples, the electronic device can generate a template for eachof the first and second hand gestures to aid in the gesture detectionprocess. For example, the electronic device can calculate first meansignal characteristics for the first cluster (e.g., as part of thek-means clustering process), and the first mean signal characteristicsmay be used as a template for the first cluster. Similarly, the firstelectronic device can calculate second mean signal characteristics forthe second cluster (e.g., as part of the k-means clustering process),and the second mean signal characteristics may be used as a template forthe second cluster. In another example, some or all of the first sensordata may be stored as the first template for the first cluster, and someor all of the second sensor data may be stored as the second templatefor the second cluster. In some examples, a generic template for eachgesture may be stored and used as a starting point for the trainingprocess before any user-specific data has been collected. Then, eachtemplate can be adjusted based on user-specific data collected duringtraining Detection using templates is discussed further below.

In some examples, additional training can be conducted to train thedevice to detect when the user is not performing either the first orsecond gesture. The electronic device can collect additional sensor dataduring a period in which the user does not perform the first or secondhand gesture. Signal characteristics can be calculated based on theadditional sensor data, and these signal characteristics can be assignedto a third cluster. The third cluster can be a cluster that isassociated with some third gesture (e.g., if the user performed a thirdgesture during the training period) or it can be a cluster that is notassociated with any gesture.

FIG. 7 illustrates an exemplary method of gesture detection inaccordance with examples of the disclosure. After training, gestures canbe detected by collecting new sensor data and then using the clustersassociated with each gesture to determine if one of the gestures hasbeen performed. During a third period (e.g., during use of the deviceafter training has concluded), the electronic device can collect (700)third sensor data from the plurality of sensors. In some examples,sensor data is further collected from sensors other than photodiodes,such as a force sensor and/or an accelerometer, among otherpossibilities.

Signal characteristics can be again extracted from the sensor data, asdescribed above with respect to FIG. 3. The electronic device cancalculate (702) the third signal characteristics based on the thirdsensor data.

To perform gesture detection, the electronic device can determine (704)whether the third signal characteristics belong to the first cluster,the second cluster, or a third cluster. The third cluster can be acluster that is associated with some third gesture, different from thefirst or second, or it can be a cluster that is not associated with anygesture.

In some examples, determining whether the third signal characteristicsbelong to the first cluster, the second cluster, or the third clusterincludes performing clustering (e.g., a k-means clustering algorithm, orother clustering algorithm) on the third signal characteristics withrespect to the first, second, and third clusters. The cluster membershipof the third signal characteristics may be determined by the results ofthe clustering.

In some examples, determining whether the third signal characteristicsbelong to the first cluster, the second cluster, or the third clusterincludes comparing the third signal characteristics to first, second,and/or third templates corresponding to the first, second, and thirdclusters, respectively. The electronic device can thereby determinewhether the third signal characteristics are closer to the first clusteror the second cluster based on the templates. For example, if eachtemplate includes mean signal characteristics, then the electronicdevice can calculate a first distance from the third signalcharacteristics to the first template (e.g., the first mean signalcharacteristics of the first cluster) and calculate a second distancefrom the third signal characteristics to the second template (e.g., thesecond mean signal characteristics of the second cluster). The distancecalculation can be a Euclidean distance calculation between two pointsin multi-dimensional space.

In accordance with a determination that the first distance is shorterthan the second distance, the electronic device can determine that thethird signal characteristics belong to the first cluster. In accordancewith a determination that the second distance is shorter than the firstdistance, the electronic device can determine that the third signalcharacteristics belong to the second cluster. In some examples, theelectronic device can also compare the third signal characteristics to athird template in the same manner, or, if both the first and seconddistances are longer than a predetermined threshold distance, theelectronic device can determine that the third signal characteristicsbelong to a third cluster by default.

Based on determining which cluster the third signal characteristicsbelong to, the electronic device can detect the first gesture, thesecond gesture, or no gesture. In accordance with a determination thatthe third signal characteristics belong to the first cluster (e.g., thecluster associated with the first hand gesture), the electronic devicecan determine (706) that the user has performed the first hand gesture.In accordance with a determination that the third signal characteristicsbelong to the second cluster (e.g., the cluster associated with thesecond hand gesture), the electronic device can determine (708) that theuser has performed the second hand gesture. In accordance with adetermination that the third signal characteristics belong to the thirdcluster (e.g., a cluster associated with some third gesture, or nogesture whatsoever), the electronic device can determine (710) that theuser has not performed the first hand gesture or the second handgesture.

After detecting the first gesture or the second gesture, the electronicdevice can perform an operation associated with the detected gesture.For example, if the electronic device detects the first gesture, theelectronic device can perform an operation in response, such as openingan application, closing an application, returning to a home screen,messaging a contact, etc.

In some examples, sensor data (e.g., the first, second, or third sensordata described above) can be further processed before extracting signalcharacteristics (e.g., the first, second, or third signalcharacteristics described above). For example, a band pass filter may beapplied to sensor data to filter out heart rate frequencies from thesensor data. As light sensor data may vary according to the periodicmotion of blood through human tissue, it may be beneficial to filter outthese frequencies to better isolate the contribution of hand gesturemotion to the signal characteristics.

Attention is now directed toward examples of portable devices withtouch-sensitive displays. FIG. 8 is a block diagram illustratingportable multifunction device 100 with touch-sensitive display system112 in accordance with some examples. Touch-sensitive display 112 issometimes called a “touch screen” for convenience and is sometimes knownas or called a “touch-sensitive display system.” Device 100 includesmemory 102 (which optionally includes one or more computer-readablestorage mediums), memory controller 122, one or more processing units(CPUs) 120, peripherals interface 118, RF circuitry 108, audio circuitry110, speaker 111, microphone 113, input/output (I/O) subsystem 106,other input control devices 116, and external port 124. Device 100optionally includes one or more optical sensors 164. Device 100optionally includes one or more contact intensity sensors 165 fordetecting intensity of contacts on device 100 (e.g., a touch-sensitivesurface such as touch-sensitive display system 112 of device 100).Device 100 optionally includes one or more tactile output generators 167for generating tactile outputs on device 100 (e.g., generating tactileoutputs on a touch-sensitive surface such as touch-sensitive displaysystem 112 of device 100 or touchpad 355 of device 300). Thesecomponents optionally communicate over one or more communication busesor signal lines 103.

As used in the specification and claims, the term “intensity” of acontact on a touch-sensitive surface refers to the force or pressure(force per unit area) of a contact (e.g., a finger contact) on thetouch-sensitive surface, or to a substitute (proxy) for the force orpressure of a contact on the touch-sensitive surface. The intensity of acontact has a range of values that includes at least four distinctvalues and more typically includes hundreds of distinct values (e.g., atleast 256). Intensity of a contact is, optionally, determined (ormeasured) using various approaches and various sensors or combinationsof sensors. For example, one or more force sensors underneath oradjacent to the touch-sensitive surface are, optionally, used to measureforce at various points on the touch-sensitive surface. In someimplementations, force measurements from multiple force sensors arecombined (e.g., a weighted average) to determine an estimated force of acontact. Similarly, a pressure-sensitive tip of a stylus is, optionally,used to determine a pressure of the stylus on the touch-sensitivesurface. Alternatively, the size of the contact area detected on thetouch-sensitive surface and/or changes thereto, the capacitance of thetouch-sensitive surface proximate to the contact and/or changes thereto,and/or the resistance of the touch-sensitive surface proximate to thecontact and/or changes thereto are, optionally, used as a substitute forthe force or pressure of the contact on the touch-sensitive surface. Insome implementations, the substitute measurements for contact force orpressure are used directly to determine whether an intensity thresholdhas been exceeded (e.g., the intensity threshold is described in unitscorresponding to the substitute measurements). In some implementations,the substitute measurements for contact force or pressure are convertedto an estimated force or pressure, and the estimated force or pressureis used to determine whether an intensity threshold has been exceeded(e.g., the intensity threshold is a pressure threshold measured in unitsof pressure). Using the intensity of a contact as an attribute of a userinput allows for user access to additional device functionality that mayotherwise not be accessible by the user on a reduced-size device withlimited real estate for displaying affordances (e.g., on atouch-sensitive display) and/or receiving user input (e.g., via atouch-sensitive display, a touch-sensitive surface, or aphysical/mechanical control such as a knob or a button).

As used in the specification and claims, the term “tactile output”refers to physical displacement of a device relative to a previousposition of the device, physical displacement of a component (e.g., atouch-sensitive surface) of a device relative to another component(e.g., housing) of the device, or displacement of the component relativeto a center of mass of the device that will be detected by a user withthe user's sense of touch. For example, in situations where the deviceor the component of the device is in contact with a surface of a userthat is sensitive to touch (e.g., a finger, palm, or other part of auser's hand), the tactile output generated by the physical displacementwill be interpreted by the user as a tactile sensation corresponding toa perceived change in physical characteristics of the device or thecomponent of the device. For example, movement of a touch-sensitivesurface (e.g., a touch-sensitive display or trackpad) is, optionally,interpreted by the user as a “down click” or “up click” of a physicalactuator button. In some cases, a user will feel a tactile sensationsuch as an “down click” or “up click” even when there is no movement ofa physical actuator button associated with the touch-sensitive surfacethat is physically pressed (e.g., displaced) by the user's movements. Asanother example, movement of the touch-sensitive surface is, optionally,interpreted or sensed by the user as “roughness” of the touch-sensitivesurface, even when there is no change in smoothness of thetouch-sensitive surface. While such interpretations of touch by a userwill be subject to the individualized sensory perceptions of the user,there are many sensory perceptions of touch that are common to a largemajority of users. Thus, when a tactile output is described ascorresponding to a particular sensory perception of a user (e.g., an “upclick,” a “down click,” “roughness”), unless otherwise stated, thegenerated tactile output corresponds to physical displacement of thedevice or a component thereof that will generate the described sensoryperception for a typical (or average) user.

It should be appreciated that device 100 is only one example of aportable multifunction device, and that device 100 optionally has moreor fewer components than shown, optionally combines two or morecomponents, or optionally has a different configuration or arrangementof the components. The various components shown in FIG. 8 areimplemented in hardware, software, or a combination of both hardware andsoftware, including one or more signal processing and/orapplication-specific integrated circuits.

Memory 102 may include one or more computer-readable storage mediums.The computer-readable storage mediums may be tangible andnon-transitory. Memory 102 may include high-speed random access memoryand may also include non-volatile memory, such as one or more magneticdisk storage devices, flash memory devices, or other non-volatilesolid-state memory devices. Memory controller 122 may control access tomemory 102 by other components of device 100.

Peripherals interface 118 can be used to couple input and outputperipherals of the device to CPU 120 and memory 102. The one or moreprocessors 120 run or execute various software programs and/or sets ofinstructions stored in memory 102 to perform various functions fordevice 100 and to process data. In some examples, peripherals interface118, CPU 120, and memory controller 122 may be implemented on a singlechip, such as chip 104. In some other examples, they may be implementedon separate chips.

RF (radio frequency) circuitry 108 receives and sends RF signals, alsocalled electromagnetic signals. RF circuitry 108 converts electricalsignals to/from electromagnetic signals and communicates withcommunications networks and other communications devices via theelectromagnetic signals. RF circuitry 108 optionally includes well-knowncircuitry for performing these functions, including but not limited toan antenna system, an RF transceiver, one or more amplifiers, a tuner,one or more oscillators, a digital signal processor, a CODEC chipset, asubscriber identity module (SIM) card, memory, and so forth. RFcircuitry 108 optionally communicates with networks, such as theInternet, also referred to as the World Wide Web (WWW), an intranetand/or a wireless network, such as a cellular telephone network, awireless local area network (LAN) and/or a metropolitan area network(MAN), and other devices by wireless communication. The RF circuitry 108optionally includes well-known circuitry for detecting near fieldcommunication (NFC) fields, such as by a short-range communicationradio. The wireless communication optionally uses any of a plurality ofcommunications standards, protocols, and technologies, including but notlimited to Global System for Mobile Communications (GSM), Enhanced DataGSM Environment (EDGE), high-speed downlink packet access (HSDPA),high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO),HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), long term evolution (LTE), nearfield communication (NFC), wideband code division multiple access(W-CDMA), code division multiple access (CDMA), time division multipleaccess (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity(Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n,and/or IEEE 802.11ac), voice over Internet Protocol (VoIP), Wi-MAX, aprotocol for e-mail (e.g., Internet message access protocol (IMAP)and/or post office protocol (POP)), instant messaging (e.g., extensiblemessaging and presence protocol (XMPP), Session Initiation Protocol forInstant Messaging and Presence Leveraging Extensions (SIMPLE), InstantMessaging and Presence Service (IMPS)), and/or Short Message Service(SMS), or any other suitable communication protocol, includingcommunication protocols not yet developed as of the filing date of thisdocument.

Audio circuitry 110, speaker 111, and microphone 113 provide an audiointerface between a user and device 100. Audio circuitry 110 receivesaudio data from peripherals interface 118, converts the audio data to anelectrical signal, and transmits the electrical signal to speaker 111.Speaker 111 converts the electrical signal to human-audible sound waves.Audio circuitry 110 also receives electrical signals converted bymicrophone 113 from sound waves. Audio circuitry 110 converts theelectrical signal to audio data and transmits the audio data toperipherals interface 118 for processing. Audio data may be retrievedfrom and/or transmitted to memory 102 and/or RF circuitry 108 byperipherals interface 118. In some examples, audio circuitry 110 alsoincludes a headset jack (e.g., 212, FIG. 2). The headset jack providesan interface between audio circuitry 110 and removable audioinput/output peripherals, such as output-only headphones or a headsetwith both output (e.g., a headphone for one or both ears) and input(e.g., a microphone).

I/O subsystem 106 couples input/output peripherals on device 100, suchas touch screen 112 and other input control devices 116, to peripheralsinterface 118. I/O subsystem 106 optionally includes display controller156, optical sensor controller 158, intensity sensor controller 159,haptic feedback controller 161, and one or more input controllers 160for other input or control devices. The one or more input controllers160 receive/send electrical signals from/to other input control devices116. The other input control devices 116 optionally include physicalbuttons (e.g., push buttons, rocker buttons, etc.), dials, sliderswitches, joysticks, click wheels, and so forth. In some alternateexamples, input controller(s) 160 are, optionally, coupled to any (ornone) of the following: a keyboard, an infrared port, a USB port, and apointer device such as a mouse. The one or more buttons (e.g., 208, FIG.2) optionally include an up/down button for volume control of speaker111 and/or microphone 113. The one or more buttons optionally include apush button (e.g., 206, FIG. 2).

A quick press of the push button may disengage a lock of touch screen112 or begin a process that uses gestures on the touch screen to unlockthe device, as described in U.S. Patent Application Pub. No.2007/0150842, “Unlocking a Device by Performing Gestures on an UnlockImage,” published Jun. 28, 2007, U.S. Pat. No. 7,657,849, which ishereby incorporated by reference in its entirety. A longer press of thepush button (e.g., 206) may turn power to device 100 on or off. The usermay be able to customize a functionality of one or more of the buttons.Touch screen 112 is used to implement virtual or soft buttons and one ormore soft keyboards.

Touch-sensitive display 112 provides an input interface and an outputinterface between the device and a user. Display controller 156 receivesand/or sends electrical signals from/to touch screen 112. Touch screen112 displays visual output to the user. The visual output may includegraphics, text, icons, video, and any combination thereof (collectivelytermed “graphics”). In some examples, some or all of the visual outputmay correspond to user-interface objects.

Touch screen 112 has a touch-sensitive surface, sensor, or set ofsensors that accepts input from the user based on haptic and/or tactilecontact. Touch screen 112 and display controller 156 (along with anyassociated modules and/or sets of instructions in memory 102) detectcontact (and any movement or breaking of the contact) on touch screen112 and convert the detected contact into interaction withuser-interface objects (e.g., one or more soft keys, icons, web pages,or images) that are displayed on touch screen 112. In one example, apoint of contact between touch screen 112 and the user corresponds to afinger of the user.

Touch screen 112 may use LCD (liquid crystal display) technology, LPD(light emitting polymer display) technology, or LED (light emittingdiode) technology, although other display technologies may be used inother examples. Touch screen 112 and display controller 156 may detectcontact and any movement or breaking thereof using any of a plurality oftouch sensing technologies now known or later developed, including butnot limited to capacitive, resistive, infrared, and surface acousticwave technologies, as well as other proximity sensor arrays or otherelements for determining one or more points of contact with touch screen112. In one example, projected mutual capacitance sensing technology isused, such as that found in the iPhone® and iPod Touch® from Apple Inc.of Cupertino, Calif.

A touch-sensitive display in some examples of touch screen 112 may beanalogous to the multi-touch sensitive touchpads described in thefollowing U.S. Pat. No. 6,323,846 (Westerman et al.), U.S. Pat. No.6,570,557 (Westerman et al.), and/or U.S. Pat. No. 6,677,932(Westerman), and/or U.S. Patent Publication 2002/0015024A1, each ofwhich is hereby incorporated by reference in its entirety. However,touch screen 112 displays visual output from device 100, whereastouch-sensitive touchpads do not provide visual output.

A touch-sensitive display in some examples of touch screen 112 may be asdescribed in the following applications: (1) U.S. Patent ApplicationPub. No. 2007/0257890, “Multipoint Touch Surface Controller,” publishedNov. 8, 2007; (2) U.S. Patent Application Pub. No. 2006/0097991,“Multipoint Touchscreen,” published May 11, 2006; (3) U.S. PatentApplication Pub. No. 2006/0026521, “Gestures For Touch Sensitive InputDevices,” published Feb. 2, 2006; (4) U.S. Patent Application Pub. No.2006/0026536, “Gestures For Touch Sensitive Input Devices,” publishedFeb. 2, 2006; (5) U.S. Patent Application Pub. No. 2006/0026535,“Mode-Based Graphical User Interfaces For Touch Sensitive InputDevices,” published Feb. 2, 2006; (6) U.S. Patent Application Pub. No.2006/0033724, “Virtual Input Device Placement On A Touch Screen UserInterface,” published Feb. 16, 2006; (7) U.S. Patent Application Pub.No. 2006/0053387, “Operation Of A Computer With A Touch ScreenInterface,” published Mar. 9, 2006; (8) U.S. Patent Application Pub. No.2006/0085757, “Activating Virtual Keys Of A Touch-Screen VirtualKeyboard,” published Apr. 20, 2006; and (9) U.S. Patent Application Pub.No. 2006/0197753, “Multi-Functional Hand-Held Device,” published Sep. 7,2006. All of these applications are incorporated by reference herein intheir entirety.

Touch screen 112 may have a video resolution in excess of 100 dpi. Insome examples, the touch screen has a video resolution of 160 dpi. Theuser may make contact with touch screen 112 using any suitable object orappendage, such as a stylus, a finger, and so forth. In some examples,the user interface is designed to work primarily with finger-basedcontacts and gestures, which can be less precise than stylus-based inputdue to the larger area of contact of a finger on the touch screen. Insome examples, the device translates the rough finger-based input into aprecise pointer/cursor position or command for performing the actionsdesired by the user.

In some examples, in addition to the touch screen, device 100 mayinclude a touchpad (not shown) for activating or deactivating particularfunctions. In some examples, the touchpad is a touch-sensitive area ofthe device that, unlike the touch screen, does not display visualoutput. The touchpad may be a touch-sensitive surface that is separatefrom touch screen 112 or an extension of the touch-sensitive surfaceformed by the touch screen.

Device 100 also includes power system 162 for powering the variouscomponents. Power system 162 may include a power management system, oneor more power sources (e.g., battery, alternating current (AC)), arecharging system, a power failure detection circuit, a power converteror inverter, a power status indicator (e.g., a light-emitting diode(LED)) and any other components associated with the generation,management and distribution of power in portable devices.

Device 100 may also include one or more optical sensors 164. FIG. 8shows an optical sensor coupled to optical sensor controller 158 in I/Osubsystem 106. Optical sensor 164 may include charge-coupled device(CCD) or complementary metal-oxide semiconductor (CMOS)phototransistors. Optical sensor 164 receives light from theenvironment, projected through one or more lenses, and converts thelight to data representing an image. In conjunction with imaging module143 (also called a camera module), optical sensor 164 may capture stillimages or video. In some examples, an optical sensor is located on theback of device 100, opposite touch screen display 112 on the front ofthe device so that the touch screen display may be used as a viewfinderfor still and/or video image acquisition. In some examples, an opticalsensor is located on the front of the device so that the user's imagemay be obtained for video conferencing while the user views the othervideo conference participants on the touch screen display. In someexamples, the position of optical sensor 164 can be changed by the user(e.g., by rotating the lens and the sensor in the device housing) sothat a single optical sensor 164 may be used along with the touch screendisplay for both video conferencing and still and/or video imageacquisition.

Device 100 optionally also includes one or more contact intensitysensors 165. FIG. 8 shows a contact intensity sensor coupled tointensity sensor controller 159 in I/O subsystem 106. Contact intensitysensor 165 optionally includes one or more piezoresistive strain gauges,capacitive force sensors, electric force sensors, piezoelectric forcesensors, optical force sensors, capacitive touch-sensitive surfaces, orother intensity sensors (e.g., sensors used to measure the force (orpressure) of a contact on a touch-sensitive surface). Contact intensitysensor 165 receives contact intensity information (e.g., pressureinformation or a proxy for pressure information) from the environment.In some examples, at least one contact intensity sensor is collocatedwith, or proximate to, a touch-sensitive surface (e.g., touch-sensitivedisplay system 112). In some examples, at least one contact intensitysensor is located on the back of device 100, opposite touch screendisplay 112, which is located on the front of device 100.

Device 100 may also include one or more proximity sensors 166. FIG. 6shows proximity sensor 166 coupled to peripherals interface 118.Alternately, proximity sensor 166 may be coupled to input controller 160in I/O subsystem 106. Proximity sensor 166 may perform as described inU.S. Patent Application Pub. Nos. 2006/0161871, “Proximity Detector InHandheld Device”; 2006/0161870, “Proximity Detector In Handheld Device”;2008/0167834, “Using Ambient Light Sensor To Augment Proximity SensorOutput”; 2007/0075965, “Automated Response To And Sensing Of UserActivity In Portable Devices”; and 2008/0140868, “Methods And SystemsFor Automatic Configuration Of Peripherals,” which are herebyincorporated by reference in their entirety. In some examples, theproximity sensor turns off and disables touch screen 112 when themultifunction device is placed near the user's ear (e.g., when the useris making a phone call).

Device 100 optionally also includes one or more tactile outputgenerators 167. FIG. 8 shows a tactile output generator coupled tohaptic feedback controller 161 in I/O subsystem 106. Tactile outputgenerator 167 optionally includes one or more electroacoustic devicessuch as speakers or other audio components and/or electromechanicaldevices that convert energy into linear motion such as a motor,solenoid, electroactive polymer, piezoelectric actuator, electrostaticactuator, or other tactile output generating component (e.g., acomponent that converts electrical signals into tactile outputs on thedevice). Contact intensity sensor 165 receives tactile feedbackgeneration instructions from haptic feedback module 133 and generatestactile outputs on device 100 that are capable of being sensed by a userof device 100. In some examples, at least one tactile output generatoris collocated with, or proximate to, a touch-sensitive surface (e.g.,touch-sensitive display system 112) and, optionally, generates a tactileoutput by moving the touch-sensitive surface vertically (e.g., in/out ofa surface of device 100) or laterally (e.g., back and forth in the sameplane as a surface of device 100). In some examples, at least onetactile output generator sensor is located on the back of device 100,opposite touch screen display 112, which is located on the front ofdevice 100.

Device 100 may also include one or more accelerometers 168. FIG. 8 showsaccelerometer 168 coupled to peripherals interface 118. Alternately,accelerometer 168 may be coupled to an input controller 160 in I/Osubsystem 106. Accelerometer 168 may perform as described in U.S. PatentPublication No. 20050190059, “Acceleration-based Theft Detection Systemfor Portable Electronic Devices,” and U.S. Patent Publication No.20060017692, “Methods And Apparatuses For Operating A Portable DeviceBased On An Accelerometer,” both of which are incorporated by referenceherein in their entirety. In some examples, information is displayed onthe touch screen display in a portrait view or a landscape view based onan analysis of data received from the one or more accelerometers. Device100 optionally includes, in addition to accelerometer(s) 168, amagnetometer (not shown) and a GPS (or GLONASS or other globalnavigation system) receiver (not shown) for obtaining informationconcerning the location and orientation (e.g., portrait or landscape) ofdevice 100.

In some examples, the software components stored in memory 102 includeoperating system 126, communication module (or set of instructions) 128,contact/motion module (or set of instructions) 130, graphics module (orset of instructions) 132, text input module (or set of instructions)134, Global Positioning System (GPS) module (or set of instructions)135, and applications (or sets of instructions) 136. Furthermore, insome examples, memory 102 (FIG. 8) or 370 (FIG. 3) stores device/globalinternal state 157, as shown in FIGS. 1A and 3. Device/global internalstate 157 includes one or more of: active application state, indicatingwhich applications, if any, are currently active; display state,indicating what applications, views or other information occupy variousregions of touch screen display 112; sensor state, including informationobtained from the device's various sensors and input control devices116; and location information concerning the device's location and/orattitude.

Operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS,WINDOWS, or an embedded operating system such as VxWorks) includesvarious software components and/or drivers for controlling and managinggeneral system tasks (e.g., memory management, storage device control,power management, etc.) and facilitates communication between varioushardware and software components.

Communication module 128 facilitates communication with other devicesover one or more external ports 124 and also includes various softwarecomponents for handling data received by RF circuitry 108 and/orexternal port 124. External port 124 (e.g., Universal Serial Bus (USB),FIREWIRE, etc.) is adapted for coupling directly to other devices orindirectly over a network (e.g., the Internet, wireless LAN, etc.). Insome examples, the external port is a multi-pin (e.g., 30-pin) connectorthat is the same as, or similar to and/or compatible with, the 30-pinconnector used on iPod® (trademark of Apple Inc.) devices.

Contact/motion module 130 optionally detects contact with touch screen112 (in conjunction with display controller 156) and othertouch-sensitive devices (e.g., a touchpad or physical click wheel).Contact/motion module 130 includes various software components forperforming various operations related to detection of contact, such asdetermining if contact has occurred (e.g., detecting a finger-downevent), determining an intensity of the contact (e.g., the force orpressure of the contact or a substitute for the force or pressure of thecontact), determining if there is movement of the contact and trackingthe movement across the touch-sensitive surface (e.g., detecting one ormore finger-dragging events), and determining if the contact has ceased(e.g., detecting a finger-up event or a break in contact).Contact/motion module 130 receives contact data from the touch-sensitivesurface. Determining movement of the point of contact, which isrepresented by a series of contact data, optionally includes determiningspeed (magnitude), velocity (magnitude and direction), and/or anacceleration (a change in magnitude and/or direction) of the point ofcontact. These operations are, optionally, applied to single contacts(e.g., one finger contacts) or to multiple simultaneous contacts (e.g.,“multitouch”/multiple finger contacts). In some examples, contact/motionmodule 130 and display controller 156 detect contact on a touchpad.

In some examples, contact/motion module 130 uses a set of one or moreintensity thresholds to determine whether an operation has beenperformed by a user (e.g., to determine whether a user has “clicked” onan icon). In some examples, at least a subset of the intensitythresholds are determined in accordance with software parameters (e.g.,the intensity thresholds are not determined by the activation thresholdsof particular physical actuators and can be adjusted without changingthe physical hardware of device 100). For example, a mouse “click”threshold of a trackpad or touch screen display can be set to any of alarge range of predefined threshold values without changing the trackpador touch screen display hardware. Additionally, in some implementations,a user of the device is provided with software settings for adjustingone or more of the set of intensity thresholds (e.g., by adjustingindividual intensity thresholds and/or by adjusting a plurality ofintensity thresholds at once with a system-level click “intensity”parameter).

Contact/motion module 130 optionally detects a gesture input by a user.Different gestures on the touch-sensitive surface have different contactpatterns (e.g., different motions, timings, and/or intensities ofdetected contacts). Thus, a gesture is, optionally, detected bydetecting a particular contact pattern. For example, detecting a fingertap gesture includes detecting a finger-down event followed by detectinga finger-up (liftoff) event at the same position as the finger-downevent (e.g., at the position of an icon). As another example, detectinga finger swipe gesture on the touch-sensitive surface includes detectinga finger-down event followed by detecting one or more finger-draggingevents, and subsequently followed by detecting a finger-up (liftoff)event.

Graphics module 132 includes various known software components forrendering and displaying graphics on touch screen 112 or other display,including components for changing the visual impact (e.g., brightness,transparency, saturation, contrast, or other visual property) ofgraphics that are displayed. As used herein, the term “graphics”includes any object that can be displayed to a user, including, withoutlimitation, text, web pages, icons (such as user-interface objectsincluding soft keys), digital images, videos, animations, and the like.

In some examples, graphics module 132 stores data representing graphicsto be used. Each graphic is, optionally, assigned a corresponding code.Graphics module 132 receives, from applications etc., one or more codesspecifying graphics to be displayed along with, if necessary, coordinatedata and other graphic property data, and then generates screen imagedata to output to display controller 156.

Haptic feedback module 133 includes various software components forgenerating instructions used by tactile output generator(s) 167 toproduce tactile outputs at one or more locations on device 100 inresponse to user interactions with device 100.

Text input module 134, which may be a component of graphics module 132,provides soft keyboards for entering text in various applications (e.g.,contacts 137, e-mail 140, IM 141, browser 147, and any other applicationthat needs text input).

GPS module 135 determines the location of the device and provides thisinformation for use in various applications (e.g., to telephone 138 foruse in location-based dialing; to camera 143 as picture/video metadata;and to applications that provide location-based services such as weatherwidgets, local yellow page widgets, and map/navigation widgets).

Applications 136 may include the following modules (or sets ofinstructions), or a subset or superset thereof:

-   -   Contacts module 137 (sometimes called an address book or contact        list);    -   Telephone module 138;    -   Video conference module 139;    -   E-mail client module 140;    -   Instant messaging (IM) module 141;    -   Workout support module 142;    -   Camera module 143 for still and/or video images;    -   Image management module 144;    -   Video player module;    -   Music player module;    -   Browser module 147;    -   Calendar module 148;    -   Widget modules 149, which may include one or more of: weather        widget 149-1, stocks widget 149-2, calculator widget 149-3,        alarm clock widget 149-4, dictionary widget 149-5, and other        widgets obtained by the user, as well as user-created widgets        149-6;    -   Widget creator module 150 for making user-created widgets 149-6;    -   Search module 151;    -   Video and music player module 152, which merges video player        module and music player module;    -   Notes module 153;    -   Map module 154; and/or    -   Online video module 155.

Examples of other applications 136 that may be stored in memory 102include other word processing applications, other image editingapplications, drawing applications, presentation applications,JAVA-enabled applications, encryption, digital rights management, voicerecognition, and voice replication.

In conjunction with touch screen 112, display controller 156,contact/motion module 130, graphics module 132, and text input module134, contacts module 137 may be used to manage an address book orcontact list (e.g., stored in application internal state 192 of contactsmodule 137 in memory 102 or memory 370), including: adding name(s) tothe address book; deleting name(s) from the address book; associatingtelephone number(s), e-mail address(es), physical address(es) or otherinformation with a name; associating an image with a name; categorizingand sorting names; providing telephone numbers or e-mail addresses toinitiate and/or facilitate communications by telephone 138, videoconference module 139, e-mail 140, or IM 141; and so forth.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111,microphone 113, touch screen 112, display controller 156, contact/motionmodule 130, graphics module 132, and text input module 134, telephonemodule 138 may be used to enter a sequence of characters correspondingto a telephone number, access one or more telephone numbers in contactsmodule 137, modify a telephone number that has been entered, dial arespective telephone number, conduct a conversation, and disconnect orhang up when the conversation is completed. As noted above, the wirelesscommunication may use any of a plurality of communications standards,protocols, and technologies.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111,microphone 113, touch screen 112, display controller 156, optical sensor164, optical sensor controller 158, contact/motion module 130, graphicsmodule 132, text input module 134, contacts module 137, and telephonemodule 138, video conference module 139 includes executable instructionsto initiate, conduct, and terminate a video conference between a userand one or more other participants in accordance with user instructions.

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact/motion module 130, graphics module 132, and textinput module 134, e-mail client module 140 includes executableinstructions to create, send, receive, and manage e-mail in response touser instructions. In conjunction with image management module 144,e-mail client module 140 makes it very easy to create and send e-mailswith still or video images taken with camera module 143.

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact/motion module 130, graphics module 132, and textinput module 134, the instant messaging module 141 includes executableinstructions to enter a sequence of characters corresponding to aninstant message, to modify previously entered characters, to transmit arespective instant message (for example, using a Short Message Service(SMS) or Multimedia Message Service (MMS) protocol for telephony-basedinstant messages or using XMPP, SIMPLE, or IMPS for Internet-basedinstant messages), to receive instant messages, and to view receivedinstant messages. In some examples, transmitted and/or received instantmessages may include graphics, photos, audio files, video files and/orother attachments as are supported in an MMS and/or an EnhancedMessaging Service (EMS). As used herein, “instant messaging” refers toboth telephony-based messages (e.g., messages sent using SMS or MMS) andInternet-based messages (e.g., messages sent using XMPP, SIMPLE, orIMPS).

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact/motion module 130, graphics module 132, textinput module 134, GPS module 135, map module 154, and music playermodule, workout support module 142 includes executable instructions tocreate workouts (e.g., with time, distance, and/or calorie burninggoals); communicate with workout sensors (sports devices); receiveworkout sensor data; calibrate sensors used to monitor a workout; selectand play music for a workout; and display, store, and transmit workoutdata.

In conjunction with touch screen 112, display controller 156, opticalsensor(s) 164, optical sensor controller 158, contact/motion module 130,graphics module 132, and image management module 144, camera module 143includes executable instructions to capture still images or video(including a video stream) and store them into memory 102, modifycharacteristics of a still image or video, or delete a still image orvideo from memory 102.

In conjunction with touch screen 112, display controller 156,contact/motion module 130, graphics module 132, text input module 134,and camera module 143, image management module 144 includes executableinstructions to arrange, modify (e.g., edit), or otherwise manipulate,label, delete, present (e.g., in a digital slide show or album), andstore still and/or video images.

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact/motion module 130, graphics module 132, and textinput module 134, browser module 147 includes executable instructions tobrowse the Internet in accordance with user instructions, includingsearching, linking to, receiving, and displaying web pages or portionsthereof, as well as attachments and other files linked to web pages.

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact/motion module 130, graphics module 132, textinput module 134, e-mail client module 140, and browser module 147,calendar module 148 includes executable instructions to create, display,modify, and store calendars and data associated with calendars (e.g.,calendar entries, to-do lists, etc.) in accordance with userinstructions.

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact/motion module 130, graphics module 132, textinput module 134, and browser module 147, widget modules 149 aremini-applications that may be downloaded and used by a user (e.g.,weather widget 149-1, stocks widget 149-2, calculator widget 149-3,alarm clock widget 149-4, and dictionary widget 149-5) or created by theuser (e.g., user-created widget 149-6). In some examples, a widgetincludes an HTML (Hypertext Markup Language) file, a CSS (CascadingStyle Sheets) file, and a JavaScript file. In some examples, a widgetincludes an XML (Extensible Markup Language) file and a JavaScript file(e.g., Yahoo! Widgets).

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact/motion module 130, graphics module 132, textinput module 134, and browser module 147, the widget creator module 150may be used by a user to create widgets (e.g., turning a user-specifiedportion of a web page into a widget).

In conjunction with touch screen 112, display controller 156,contact/motion module 130, graphics module 132, and text input module134, search module 151 includes executable instructions to search fortext, music, sound, image, video, and/or other files in memory 102 thatmatch one or more search criteria (e.g., one or more user-specifiedsearch terms) in accordance with user instructions.

In conjunction with touch screen 112, display controller 156,contact/motion module 130, graphics module 132, audio circuitry 110,speaker 111, RF circuitry 108, and browser module 147, video and musicplayer module 152 includes executable instructions that allow the userto download and play back recorded music and other sound files stored inone or more file formats, such as MP3 or AAC files, and executableinstructions to display, present, or otherwise play back videos (e.g.,on touch screen 112 or on an external, connected display via externalport 124). In some examples, device 100 optionally includes thefunctionality of an MP3 player, such as an iPod (trademark of AppleInc.).

In conjunction with touch screen 112, display controller 156,contact/motion module 130, graphics module 132, and text input module134, notes module 153 includes executable instructions to create andmanage notes, to-do lists, and the like in accordance with userinstructions.

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact/motion module 130, graphics module 132, textinput module 134, GPS module 135, and browser module 147, map module 154may be used to receive, display, modify, and store maps and dataassociated with maps (e.g., driving directions, data on stores and otherpoints of interest at or near a particular location, and otherlocation-based data) in accordance with user instructions.

In conjunction with touch screen 112, display controller 156,contact/motion module 130, graphics module 132, audio circuitry 110,speaker 111, RF circuitry 108, text input module 134, e-mail clientmodule 140, and browser module 147, online video module 155 includesinstructions that allow the user to access, browse, receive (e.g., bystreaming and/or download), play back (e.g., on the touch screen or onan external, connected display via external port 124), send an e-mailwith a link to a particular online video, and otherwise manage onlinevideos in one or more file formats, such as H.264. In some examples,instant messaging module 141, rather than e-mail client module 140, isused to send a link to a particular online video. Additional descriptionof the online video application can be found in U.S. Provisional PatentApplication No. 60/936,562, “Portable Multifunction Device, Method, andGraphical User Interface for Playing Online Videos,” filed Jun. 20,2007, and U.S. Patent Application Pub. No. 2008/0320391, “PortableMultifunction Device, Method, and Graphical User Interface for PlayingOnline Videos,” published Dec. 25, 2008, the contents of which arehereby incorporated by reference in their entirety.

Each of the above-identified modules and applications corresponds to aset of executable instructions for performing one or more functionsdescribed above and the methods described in this application (e.g., thecomputer-implemented methods and other information processing methodsdescribed herein). These modules (e.g., sets of instructions) need notbe implemented as separate software programs, procedures, or modules,and thus various subsets of these modules may be combined or otherwiserearranged in various examples. For example, video player module may becombined with music player module into a single module (e.g., video andmusic player module 152, FIG. 8). In some examples, memory 102 may storea subset of the modules and data structures identified above.Furthermore, memory 102 may store additional modules and data structuresnot described above.

In some examples, device 100 is a device where operation of a predefinedset of functions on the device is performed exclusively through a touchscreen and/or a touchpad. By using a touch screen and/or a touchpad asthe primary input control device for operation of device 100, the numberof physical input control devices (such as push buttons, dials, and thelike) on device 100 may be reduced.

The predefined set of functions that are performed exclusively through atouch screen and/or a touchpad optionally include navigation betweenuser interfaces. In some examples, the touchpad, when touched by theuser, navigates device 100 to a main, home, or root menu from any userinterface that is displayed on device 100. In such examples, a “menubutton” is implemented using a touchpad. In some other examples, themenu button is a physical push button or other physical input controldevice instead of a touchpad.

In some examples, a method of detecting hand gestures performed by auser of an electronic device including a plurality of photodiodes isdisclosed. The method may include: during a first period in which theuser performs a first hand gesture, collecting first sensor data fromthe plurality of photodiodes; during a second period in which the userperforms a second hand gesture, collecting second sensor data from theplurality of photodiodes; calculating first signal characteristics basedon the first sensor data and second signal characteristics based on thesecond sensor data; assigning some or all of the first signalcharacteristics to a first cluster of signal characteristics; assigningsome or all of the second signal characteristics to a second cluster ofsignal characteristics; during a third period, collecting third sensordata from the plurality of photodiodes; calculating third signalcharacteristics based on the third sensor data; determining whether thethird signal characteristics belong to the first cluster, the secondcluster, or a third cluster; in accordance with a determination that thethird signal characteristics belong to the first cluster, determiningthat the user has performed the first hand gesture; in accordance with adetermination that the third signal characteristics belong to the secondcluster, determining that the user has performed the second handgesture; and in accordance with a determination that the third signalcharacteristics belong to the third cluster, determining that the userhas not performed the first hand gesture or the second hand gesture.

Additionally or alternatively to the other disclosed examples, themethod may further include: comparing the first cluster to the secondcluster; based on comparing the first cluster to the second cluster,determining there are more of the first signal characteristics assignedto the first cluster than to the second cluster; and in accordance withthe determination that there are more of the first signalcharacteristics assigned to the first cluster than to the secondcluster, assigning the first cluster to the first hand gesture.Additionally or alternatively to the other disclosed examples, themethod may further include: based on comparing the first cluster to thesecond cluster, determining there are more of the second signalcharacteristics assigned to the second cluster than to the firstcluster; and in accordance with the determination that there are more ofthe second signal characteristics assigned to the second cluster than tothe first cluster, assigning the second cluster to the second handgesture. Additionally or alternatively to the other disclosed examples,the method may further include: generating a first template for thefirst hand gesture and a second template for the second hand gesture;and comparing the third signal characteristics to the first and secondtemplates. Additionally or alternatively to the other disclosedexamples, the method may further include: calculating first mean signalcharacteristics for the first cluster, wherein the first template isgenerated based on the first mean signal characteristics for the firstcluster; and calculating second mean signal characteristics for thesecond cluster, wherein the second template is generated based on thesecond mean signal characteristics for the second cluster. Additionallyor alternatively to the other disclosed examples, comparing the thirdsignal characteristics to the first and second templates may include:calculating a first distance from the third signal characteristics tothe first template and calculating a second distance from the thirdsignal characteristics to the second template; in accordance with adetermination that the first distance is shorter than the seconddistance, determining that the third signal characteristics belong tothe first cluster; and in accordance with a determination that thesecond distance is shorter than the first distance, determining that thethird signal characteristics belong to the second cluster. Additionallyor alternatively to the other disclosed examples, generating the firsttemplate for the first hand gesture may include storing some or all ofthe first sensor data as the first template, and generating the secondtemplate for the second hand gesture may include storing some or all ofthe second sensor data as the second template. Additionally oralternatively to the other disclosed examples, calculating the firstsignal characteristics may include calculating at least one of: anamplitude difference between a peak and a trough of the first sensordata, a time difference between a peak and a trough of the first sensordata, a maximum amplitude of the first sensor data, a period betweenpeaks of the first sensor data, and a phase of the first sensor data.Additionally or alternatively to the other disclosed examples, themethod may further include: filtering heart rate frequencies from thefirst sensor data before calculating the first signal characteristicsbased on the first sensor data. Additionally or alternatively to theother disclosed examples, the method may further include: during thefirst period in which the user performs the first hand gesture, furthercollecting the first sensor data from at least one of a force sensor andan accelerometer. Additionally or alternatively to the other disclosedexamples, collecting the first sensor data from the plurality ofphotodiodes may include collecting a first channel of infrared light andcollecting a second channel of green light. Additionally oralternatively to the other disclosed examples, the method may furtherinclude: during a fourth period in which the user does not perform thefirst or second hand gesture, collecting fourth sensor data from theplurality of photodiodes; calculating fourth signal characteristicsbased on the fourth sensor data; and assigning some or all of the fourthsignal characteristics to the third cluster of signal characteristics.Additionally or alternatively to the other disclosed examples, assigningthe first and second signal characteristics to the first and secondclusters may be performed using a k-means clustering algorithmAdditionally or alternatively to the other disclosed examples, thek-means clustering algorithm may also be applied to the third signalcharacteristics and determining whether the third signal characteristicsbelong to the first cluster, the second cluster, or the third clustermay be based on the k-means clustering algorithm.

In some examples, a non-transitory computer readable medium isdisclosed, the computer readable medium containing instructions, that,when executed, perform a method. The method may include: The method mayinclude: during a first period in which the user performs a first handgesture, collecting first sensor data from the plurality of photodiodes;during a second period in which the user performs a second hand gesture,collecting second sensor data from the plurality of photodiodes;calculating first signal characteristics based on the first sensor dataand second signal characteristics based on the second sensor data;assigning some or all of the first signal characteristics to a firstcluster of signal characteristics; assigning some or all of the secondsignal characteristics to a second cluster of signal characteristics;during a third period, collecting third sensor data from the pluralityof photodiodes; calculating third signal characteristics based on thethird sensor data; determining whether the third signal characteristicsbelong to the first cluster, the second cluster, or a third cluster; inaccordance with a determination that the third signal characteristicsbelong to the first cluster, determining that the user has performed thefirst hand gesture; in accordance with a determination that the thirdsignal characteristics belong to the second cluster, determining thatthe user has performed the second hand gesture; and in accordance with adetermination that the third signal characteristics belong to the thirdcluster, determining that the user has not performed the first handgesture or the second hand gesture.

In some examples, an electronic device is disclosed. The electronicdevice may include: one or more processors; memory; a plurality ofphotodiodes; and one or more programs, wherein the one or more programsare stored in the memory and are configured to be executed by the one ormore processors, which when executed by the one or more processors,cause the electronic device to perform a method. The method may include:during a first period in which the user performs a first hand gesture,collecting first sensor data from the plurality of photodiodes; during asecond period in which the user performs a second hand gesture,collecting second sensor data from the plurality of photodiodes;calculating first signal characteristics based on the first sensor dataand second signal characteristics based on the second sensor data;assigning some or all of the first signal characteristics to a firstcluster of signal characteristics; assigning some or all of the secondsignal characteristics to a second cluster of signal characteristics;during a third period, collecting third sensor data from the pluralityof photodiodes; calculating third signal characteristics based on thethird sensor data; determining whether the third signal characteristicsbelong to the first cluster, the second cluster, or a third cluster; inaccordance with a determination that the third signal characteristicsbelong to the first cluster, determining that the user has performed thefirst hand gesture; in accordance with a determination that the thirdsignal characteristics belong to the second cluster, determining thatthe user has performed the second hand gesture; and in accordance with adetermination that the third signal characteristics belong to the thirdcluster, determining that the user has not performed the first handgesture or the second hand gesture.

Although the disclosed examples have been fully described with referenceto the accompanying drawings, it is to be noted that various changes andmodifications will become apparent to those skilled in the art. Suchchanges and modifications are to be understood as being included withinthe scope of the disclosed examples as defined by the appended claims.

1. A method of detecting hand gestures performed by a user of anelectronic device including a plurality of photodiodes, the methodcomprising: during a first period in which the user performs a firsthand gesture, collecting first sensor data from the plurality ofphotodiodes; during a second period in which the user performs a secondhand gesture, collecting second sensor data from the plurality ofphotodiodes; calculating first signal characteristics based on the firstsensor data and second signal characteristics based on the second sensordata; assigning some or all of the first signal characteristics to afirst cluster of signal characteristics; assigning some or all of thesecond signal characteristics to a second cluster of signalcharacteristics; during a third period, collecting third sensor datafrom the plurality of photodiodes; calculating third signalcharacteristics based on the third sensor data; determining whether thethird signal characteristics belong to the first cluster, the secondcluster, or a third cluster; in accordance with a determination that thethird signal characteristics belong to the first cluster, determiningthat the user has performed the first hand gesture; in accordance with adetermination that the third signal characteristics belong to the secondcluster, determining that the user has performed the second handgesture; and in accordance with a determination that the third signalcharacteristics belong to the third cluster, determining that the userhas not performed the first hand gesture or the second hand gesture. 2.The method of claim 1, the method further comprising: comparing thefirst cluster to the second cluster; based on comparing the firstcluster to the second cluster, determining there are more of the firstsignal characteristics assigned to the first cluster than to the secondcluster; and in accordance with the determination that there are more ofthe first signal characteristics assigned to the first cluster than tothe second cluster, assigning the first cluster to the first handgesture.
 3. The method of claim 2, the method further comprising: basedon comparing the first cluster to the second cluster, determining thereare more of the second signal characteristics assigned to the secondcluster than to the first cluster; and in accordance with thedetermination that there are more of the second signal characteristicsassigned to the second cluster than to the first cluster, assigning thesecond cluster to the second hand gesture.
 4. The method of claim 1, themethod further comprising: generating a first template for the firsthand gesture and a second template for the second hand gesture; andcomparing the third signal characteristics to the first and secondtemplates.
 5. The method of claim 4, the method further comprising:calculating first mean signal characteristics for the first cluster,wherein the first template is generated based on the first mean signalcharacteristics for the first cluster; and calculating second meansignal characteristics for the second cluster, wherein the secondtemplate is generated based on the second mean signal characteristicsfor the second cluster.
 6. The method of claim 5, wherein comparing thethird signal characteristics to the first and second templates includes:calculating a first distance from the third signal characteristics tothe first template and calculating a second distance from the thirdsignal characteristics to the second template; in accordance with adetermination that the first distance is shorter than the seconddistance, determining that the third signal characteristics belong tothe first cluster; and in accordance with a determination that thesecond distance is shorter than the first distance, determining that thethird signal characteristics belong to the second cluster.
 7. The methodof claim 4, wherein generating the first template for the first handgesture includes storing some or all of the first sensor data as thefirst template, and generating the second template for the second handgesture includes storing some or all of the second sensor data as thesecond template.
 8. The method of claim 1, wherein calculating the firstsignal characteristics includes calculating at least one of: anamplitude difference between a peak and a trough of the first sensordata, a time difference between a peak and a trough of the first sensordata, a maximum amplitude of the first sensor data, a period betweenpeaks of the first sensor data, and a phase of the first sensor data. 9.The method of claim 1, the method further comprising: filtering heartrate frequencies from the first sensor data before calculating the firstsignal characteristics based on the first sensor data.
 10. The method ofclaim 1, the method further comprising: during the first period in whichthe user performs the first hand gesture, further collecting the firstsensor data from at least one of a force sensor and an accelerometer.11. The method of claim 1, wherein collecting the first sensor data fromthe plurality of photodiodes includes collecting a first channel ofinfrared light and collecting a second channel of green light.
 12. Themethod of claim 1, the method further comprising: during a fourth periodin which the user does not perform the first or second hand gesture,collecting fourth sensor data from the plurality of photodiodes;calculating fourth signal characteristics based on the fourth sensordata; and assigning some or all of the fourth signal characteristics tothe third cluster of signal characteristics.
 13. The method of claim 1,wherein assigning the first and second signal characteristics to thefirst and second clusters is performed using a k-means clusteringalgorithm.
 14. The method of claim 13, wherein the k-means clusteringalgorithm is also applied to the third signal characteristics anddetermining whether the third signal characteristics belong to the firstcluster, the second cluster, or the third cluster is based on thek-means clustering algorithm.
 15. A non-transitory computer readablemedium, the computer readable medium containing instructions, that, whenexecuted, perform a method comprising: during a first period in whichthe user performs a first hand gesture, collecting first sensor datafrom a plurality of photodiodes; during a second period in which theuser performs a second hand gesture, collecting second sensor data fromthe plurality of photodiodes; calculating first signal characteristicsbased on the first sensor data and second signal characteristics basedon the second sensor data; assigning some or all of the first signalcharacteristics to a first cluster of signal characteristics; assigningsome or all of the second signal characteristics to a second cluster ofsignal characteristics; during a third period, collecting third sensordata from the plurality of sensors; calculating third signalcharacteristics based on the third sensor data; determining whether thethird signal characteristics belong to the first cluster, the secondcluster, or a third cluster; in accordance with a determination that thethird signal characteristics belong to the first cluster, determiningthat the user has performed the first hand gesture; in accordance with adetermination that the third signal characteristics belong to the secondcluster, determining that the user has performed the second handgesture; and in accordance with a determination that the third signalcharacteristics belong to the third cluster, determining that the userhas not performed the first hand gesture or the second hand gesture. 16.The non-transitory computer readable medium of claim 15, the methodfurther comprising: comparing the first cluster to the second cluster;based on comparing the first cluster to the second cluster, determiningthere are more of the first signal characteristics assigned to the firstcluster than to the second cluster; and in accordance with thedetermination that there are more of the first signal characteristicsassigned to the first cluster than to the second cluster, assigning thefirst cluster to the first hand gesture.
 17. The non-transitory computerreadable medium of claim 16, the method further comprising: based oncomparing the first cluster to the second cluster, determining there aremore of the second signal characteristics assigned to the second clusterthan to the first cluster; and in accordance with the determination thatthere are more of the second signal characteristics assigned to thesecond cluster than to the first cluster, assigning the second clusterto the second hand gesture.
 18. The non-transitory computer readablemedium of claim 15, the method further comprising: generating a firsttemplate for the first hand gesture and a second template for the secondhand gesture; and comparing the third signal characteristics to thefirst and second templates.
 19. The non-transitory computer readablemedium of claim 18, the method further comprising: calculating firstmean signal characteristics for the first cluster, wherein the firsttemplate is generated based on the first mean signal characteristics forthe first cluster; and calculating second mean signal characteristicsfor the second cluster, wherein the second template is generated basedon the second mean signal characteristics for the second cluster.
 20. Anelectronic device comprising: one or more processors; memory; aplurality of photodiodes; and one or more programs, wherein the one ormore programs are stored in the memory and are configured to be executedby the one or more processors, which when executed by the one or moreprocessors, cause the electronic device to perform a method comprising:during a first period in which the user performs a first hand gesture,collecting first sensor data from the plurality of photodiodes; during asecond period in which the user performs a second hand gesture,collecting second sensor data from the plurality of photodiodes;calculating first signal characteristics based on the first sensor dataand second signal characteristics based on the second sensor data;assigning some or all of the first signal characteristics to a firstcluster of signal characteristics; assigning some or all of the secondsignal characteristics to a second cluster of signal characteristics;during a third period, collecting third sensor data from the pluralityof sensors; calculating third signal characteristics based on the thirdsensor data; determining whether the third signal characteristics belongto the first cluster, the second cluster, or a third cluster; inaccordance with a determination that the third signal characteristicsbelong to the first cluster, determining that the user has performed thefirst hand gesture; in accordance with a determination that the thirdsignal characteristics belong to the second cluster, determining thatthe user has performed the second hand gesture; and in accordance with adetermination that the third signal characteristics belong to the thirdcluster, determining that the user has not performed the first handgesture or the second hand gesture.